feat(evaluations)!: route handlers by provider and mode - #152

Summary

This change implements the handler routing rules of the shared spec, TESTING.md §8, in launchdarkly/ai-sdks-monorepo#52. It lets one evaluation run use different providers or modes for generation and for judges. For example, generation uses OpenAI and a judge uses Anthropic, or generation uses an agent handler and a judge uses a messages handler.

This is a breaking change to the experimental evaluations API.

handler or handlers

• run() takes one handler as handler or a list as handlers, not both. judge_handlers is removed, with no alias.
• Every handler must declare provides_for. Before this change, a handler without it received every judge config with no warning.
• Two handlers with the same provider and mode raise an error.

One selection rule

• Generation and every judge select a handler with utils.select_handler, the same rule as config(). The evals code has no copy of it.
• There is no mode fallback. A messages-mode judge never runs on an agent handler, so collapse_messages is removed from the evals path. The online judges.py path does not change.

generation.mode

• mode is required: "completion" or "agent". There is no default.
• Other values, such as "messages", "judge" or a typo, raise before any network I/O. They do not go through normalize_mode.
• GenerationConfig stays a TypedDict with Required[...], so callers pass a plain dict and import no type. The new GenerationOverrides TypedDict is the type for overrides with ai_config. Two @overload signatures on run() select between them.
• The GenerationConfig docstring and the README tell the user how to choose a mode.
• The mode is not sent to LaunchDarkly.

Prompt must match the handler mode

• A messages handler with instructions raises: "the generation prompt needs messages".
• An agent handler with messages raises: "the generation prompt needs instructions".
• The same check applies to each judge. All judge problems are reported in one error, before any record is created.

Judges get no tools

A judge call gets an empty tool map, and tools is removed from the judge config.

ai_config

• The run first reads GET ai-configs/<key> to get the mode. The mode is on the AI Config, not on the variation.
• Then it reads the variation, and then the model config.
• Each check runs right after the read that supplies its input:
  • mode coverage: after 1 GET
  • prompt: after 2 GETs
  • provider: after the model config read
• The mode selects the variation's prompt field: instructions in agent mode, messages in completion and judge mode.
• When the mode's field is empty, the other field is kept, so that the prompt check rejects it.
• Overrides cannot set mode.

Migration

# Before
await evals.run(..., handler=openai_handler, judge_handlers=[claude_handler],
                generation={"provider": "OpenAI", "model": "gpt-4o"})

# After
await evals.run(..., handlers=[openai_handler, claude_handler],
                generation={"provider": "OpenAI", "model": "gpt-4o", "mode": "completion"})

To evaluate your own code, wrap it: create_handler(("OpenAI", "messages"), my_app).

Tests

• The existing tests now use tagged handlers and set mode. The judge fixtures now use messages, which is what judge configs carry.
• New tests cover:
  • handler argument validation
  • mode validation
  • prompt and mode mismatches
  • selection by provider and mode, and exact provider over wildcard
  • no mode fallback for judges
  • one error for all judge problems
  • judges with no tools
  • the request count at each ai_config failure
  • prompt field selection by mode
• make test after the merge with main: 2583 passed, 14 skipped. make lint, make format-check and make typecheck are clean.

Review fixes

• utils.select_handler now compares a handler's provider and its mode after normalize_mode. Before this change, a handler tagged ("OpenAI", "completion") or with a list passed validation but was never selected. The fix is in the shared function, so the online config() path is fixed too. Tests for both paths fail without the fix.
• GenerationMode and GenerationOverrides are exported from launchdarkly_ai_server.experimental.evaluations, which is the public path since feat(evaluations)!: mark evaluations from code as experimental #157. They are also in its surface test.
• The branch is merged with main. The packages/ai/README.md conflict takes the side of main.

Out of scope

• The spec's tools map shape and its GET ai-tools step at run time. Python uses list[EvalTool] and tools.get(), which was an earlier divergence.

BEGIN_COMMIT_OVERRIDE
feat(evaluations)!: route handlers by provider and mode
fix(client): match a handler's provides_for after normalizing its mode
END_COMMIT_OVERRIDE

🤖 Generated with Claude Code